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Abstract 

Restricted Isometry Constants (RICs) provide a measure of how far from an 
isometry a matrix can be when acting on sparse vectors. This, and related 
quantities, provide a mechanism by which standard eig en- analysis can be applied 
to topics relying on sparsity. RIC bounds have been presented for a variety 
of random matrices and matrix dimension and sparsity ranges. We provide 
explicitly formulae for RIC bounds, of n x N Gaussian matrices with sparsity 
k, in three settings: a) n/N fixed and k/n approaching zero, b) k/n fixed and 
n/N approaching zero, and c) n/N approaching zero with k/n decaying inverse 
logrithmically in N/n; in these three settings the RICs a) decay to zero, b) 
become unbounded (or approach inherent bounds), and c) approach a non-zero 
constant. Implications of these results for RIC based analysis of compressed 
sensing algorithms are presented. 

Keywords: restricted isometry constant, Gaussian matrices, singular values of 
random matrices, compressed sensing, sparse approximation 
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1. Introduction 

Many questions in signal processing[l, 2], statistics [3, 4, 5], computer vision 
[6, 7, 8, 9], and machine learning [10, 11, 12] are employing a parsimonious no- 
tion of eigen- analysis to better capture inherent simplicity in the data. Slight 
variants of the same quantity are defined in these disciplines, referred to as: 
sparse principal components, sparse eigenvalues, and restricted isometry con- 
stants (RICs). In this article we adopt the notation and terminology of RICs, 
defined as a measure of the greatest relative change that a matrix can induce in 
the £ 2 norm of sparse vectors. Let X (fc) denote all vectors of length N which 
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have at most k nonzeros; then the lower and upper RICs of the n x N matrix 
A are defined as 



L(k, n, N; A) 



xe X N (k) \\X\\$ 




and 



(1) 



U{k,n,N;A) 




1 respectively. 



(2) 



RICs were introduced by Candes and Tao in 2004 [13] as a method of analysis for 
sparse approximation and compressed sensing (CS), and has received widespread 
used in those communities. For example, let y = Ax + e for some x € X N {k), 
then, provided the RICs of A are sufficiently small, there are computationally 
tractable algorithms which from A and y (and possibly k and ||e||) are guaran- 
teed to return a vector x satisfying a bound of the form ||a;o — £||2 < Const. 1 1 e 1 1 2 ; 
for examples of such theorems see [14, 15, 16, 17, 18, 19]. The efficacy of theo- 
rems of this form depends highly on knowledge of the RICs of A. 

Numerous algorithms exist for estimating or bounding the RICs of a general 
matrix; however, theory for the current state of the art [20, 21] is limited to 
k <~ y/n, whereas many applications require information for comparatively larger 
values of k. The only method for calculating the RICs of a general matrix A 
for larger values of k, requires calculating the extreme singular values of all (^f) 
submatrices of A, resulting from all independent selections of k columns from 
A. This combinatorial approach is intractable for all but very small dimensions. 
For this reason, much of the research on RICs has been devoted to deriving their 
bounds. Matrices with entries drawn from the Gaussian distribution K(0, 1/n) 
have the smallest known bound for large matrices and k » 1 [22]. For bounds 
on the RICs of matrix ensembles other than Gaussian see [23, 24, 25]. 



RIC bounds for Gaussian matrices have been derived focusing on the limits 
p n — > p € (0, 1) and <5„ — > S e (0, 1), [22, 26, 13], see Theorem 1. Unfortunately, 
these bounds are given in terms of implicitly defined functions, Definition 8, 
obscuring their dependence on p and 8. 

Theorem 1 (Gaussian RIC Bounds [26]). Let L(S,p) andll(5,p) be defined 
as in Definition 8 and fix e > 0. In the limit where n/N — > 5 e (0,1) and 
k/n — > p G (0,1) as n — > 00, sample each n x N matrix A from the Gaus- 
sian ensemble ( entries drawn independent and identically distributed from the 
Gaussian Normal N(0, 1/n)) then 



exponentially in n. 

In this manuscript we present simple expressions which bound the RICs of 
Gaussian matrices in three asymptotic settings: (a) S e (0, 1) and p< 1 where 



Let 



k 



and 6 n := —. 



Pn ■= - 

n 



Prob(L(k,n,N;A) <L(S,p) + e) ->• 1 and 
Prob (U(k, n, N;A)<U (5, p) + e) -> 1 
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the RICs converge to zero as p approaches zero, (b) p £ (0, 1) and <5< 1 where 
the upper RIC become unbounded and the lower RIC converges to its bound 
of one as 5 approaches zero, and (c) along the path p 7 (<5) = 1/ (7log(£ -1 )) for 
5 <C 1 where the RICs approach a nonzero constant as 5 approaches zero. 

The bounds presented here build on the results in [26] and are specific to 
Gaussian matrices, carefully balancing combinatorial quantities with the tail 
behaviour of the largest and smallest singular values of Gaussian matrices. The 
specificity of these bounds to Gaussian matrices gives great accuracy than what 
subgaussian tail bounds provide [27]. A similar analysis could be conducted for 
the subgaussian case by considering the bounds in [13] stated for the Gaussian 
case, but which are equally valid for the subgaussian case. For brevity we do 
not consider the subgaussian case here. 

Each of Theorems 2-4 state that the probability under consideration con- 
verge exponentially to 1 in k or n which we use as a shorthand for saying one 
minus the probability considered being bounded by a function decaying expo- 
nentially to zero in the variable stated; the explicit bound is given in the proof 
of the theorem. 

Theorem 1 states that, for k, n, and N large, it is unlikely that the RICs 
exceed the constants £ (5, p) and U [5, p) by more than any e. In the limit where 
S n — » S £ (0, 1) and p n — > p <C 1, the matrix RICs converge to zero, causing the 
resulting bounds to become vacuous. Theorem 2 states the dominant terms in 
the bounds, and that the true RICs are unlikely to exceed these bounds by a 
multiplicative factor (1+e) for any e > 0. The dominant terms can be contrasted 
with 2^/p+p which is the deviation from one of the expected value of the smallest 
and largest eigenvalues of a Wishart matrix [28, 29]. An implication of Theorem 
2 for the compressed sensing algorithm Orthogonal Matching Pursuit is given 
in Corollary 7. 

Theorem 2 (Gaussian RIC Bounds: p <C 1). Let U p (5,p) and L p (S,p) be 
defined as 



U"UM= (-t^t) +■<■/, (3) 



LP(5,p) = ] j2p\og^—)+cp. (4) 

Fix e > and c > 6. For each S £ (0, 1) there exists a p > such that in the 
limit where n/N — > 5, k/n — > p £ (0,po); an d (logn)/fc — > as k — > oo ; sample 
each n x N matrix A from the Gaussian ensemble, INT (0, l/n), then 

Prob (L(k, n, N; A) < (1 + e)l p (5, p)j 1 and 

Prob (ll(k, n, N; A) < (1 + e)U p (5, p)) -> 1 

exponentially in k. 
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Theorem 3 considers a limiting case where the upper RIC diverges and the 
lower RIC converges to its bound of one. With the lower RIC converging to 
one, its bound is shown to be no more than an arbitrarily small multiplicative 
constant, whereas the upper RIC is bounded by an additive constant. 

Theorem 3 (Gaussian RIC Bounds: S < 1). Let U s (S,p) and Z 6 ''(5, p) be 
defined as 



U S (S, P ) 
Z s (6,p) 



p\og 



S 2 p s 



+ (1 + p) log 



clog 



1 



S 2 p i 



+ 3p, 



1 — cxp 



3/7 + c 



(5) 
(6) 



Fix e > and c > 1. For each p G (0, 1) there exists a 5q > such that in the 
limit where k/n — > p, n/N — > 5 € (0,<5o) as n — > oo, sample each n x N matrix 
A from the Gaussian ensemble, N(0, 1/n), then 



Prob (L(k, n, N; A) < (1 + e)L 5 (S, p) 
Prob (U(k, n, N; A) < U s (5, p) + e 



1 and 



exponentially in n. 

Theorem 4 considers the path in which both p n and S n converge to zero, 
but in such a way that the RICs approach nonzero constants. This path is of 
particular interest in applications where RICs are required to remain bounded, 
but where the most extreme advantages of the method are achieved for one 
of the quantities approaching zero. For example, compressed sensing achieves 
increased gains in undersampling as S n decreases to zero; however, all com- 
presses sensing algorithmic guarantees involving RICs require the RICs to re- 
main bounded. The limit considered in Theorem 4 provides explicit formula for 
these algorithms in the case where the undersampling is greatest, see Corollary 
6. 

Theorem 4 (Gaussian RIC Bounds: p n — > (7log(l/<j~„)) _1 and S <C 1). Let 

Pfify = 7 iog((5- 1 ) an d ^ ^ 7 Pt(fi)) an d ^ 7 (^i Pt(fi)) be defined as 



UT(J,p 7 (J)) = i/2p 7 (5)Iog 



1 



s 2 P m 



+ c u 



6p 7 (c5) 



2p 7 (<5)log 



s 2 P m 



6p 7 (S) 



(7) 



W (S,p 7 (8)) = J 2p 7 (5) log 



s 2 P m 



- Q 



6p 7 ((5) 



p^(S) log 



s 2 P m 



+ 6p 7 (5) 



(8) 
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Fix 7 > 70 (which 70 > 4), e > 7 c u > 1/3 and ci < 1/3. There exists a 
So > suc/i i/iai m the limit where k/n — > /9 7 (oo), n/7V — > <5 £ (0, #0) as n — > 00 , 
sample each n x N matrix A from the Gaussian ensemble, K(0, 1/n), £/ien 

Pro& (L(fc, n, AT; A) < £ 7 (<5, p 7 (oj) + e) -> 1 and 
Pro& (V(fc, n, N; A) < VJ (5, p 7 (S)) + e) -> 1 

exponentially in n. 

Theorem 4 considers the path /9 7 (<S) for (5 <C 1; passing to the limit of 6 — > 0, 
the functions U 7 (5, p 7 (oj) and £ 7 (<5, p 7 (oj) defined as (7) and (8) converge to 
simple functions of 7. 

Corollary 5 (Gaussian RIC Bounds: p n — > (7log(l/<5„))~ 1 as <5 — > 0). Let 

U 7 (<$, p 7 (<$)) and L" 1 (6,p 7 (6)) be defined as (7) and (8) respectively with p 7 (S) = 
1 

lim it 7 (W<5)) = ^= + -c« (9) 

limZ 7 (<5,p 7 (<5)) - ^--- Cl . (10) 

V7 7 



5->0 



The accuracy of Theorems 2-4 and Corollary 5 are discussed in Section 2 
and proven in Section 3. 



1.1. Compressed sensing sampling theorems 

Compressed sensing is a technique by which simplicity in data can be ex- 
ploited to reduce the amount of measurements needed to acquire the data. For 
example, let there be a vector xo <E X W (fc) which satisfies y = Axo + e; the ma- 
trix A can be viewed as measuring xq through inner products between its rows 
and xo, and e captures the model misfit such as measurement error or the true 
measured vector not being exactly k sparse. If we let A be of size n x N with 
n < N, then fewer than N inner products have been performed, and naively it 
seems impossible to recover xo- 

The theory of compressed sensing has developed conditions in which xo, or 
an approximation thereof, can be recovered. Most remarkably, for any fixed 
ratio n/N, the recovery guarantees achieve the optimal order of the number of 
measurements being proportional to the information content in xq (n propor- 
tional to k). In fact, for most compressed sensing algorithms it is possible to 
derive constants of proportionality, p al9 (6), such that if A has entries N(0, 1/n), 
then in the limit of n -> 00 with n/N -> S e (0, 1) and k/n < (1 - e)p al9 {5) it 
can be guaranteed that the output of a compressed sensing algorithm, x, will 
satisfy \\xo — x\\ 2 < Const. 1 1 e 1 1 2 . The best current known values of p alg {8) have 
been calculated in [14] for Iterative Hard Thresholding (IHT) [15], Subspace 
Pursuit (SP) [18], and Compressed Sampling Matching Pursuit (CoSaMP) [19]. 
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Compressed sensing is most remarkable in that the recovery algorithms re- 
main effective for k/n decaying slowly as the number of measurements becomes 
vanishingly small compared to the signal length, n/N — > 0. In fact, it is known 
that p al9 {5) becomes proportional to 1/ (log(<5 -1 )) as 5 — > 0. This constant of 
proportionality can be deduced from Theorem 4; the resulting sampling theo- 
rems for representative compressed sensing algorithms are stated in Corollary 6 
for c u = q = 1/3. 

Corollary 6. Given a sensing matrix, A, of size n x N whose entries are 
drawn i.i.d. from N(0, in the limit as n/N — > a sufficient condition for 
recovery for Compressed Sensing algorithms is n > jk log [N/n) measurements 
with 7 = 37 for 1 i -minimization [16], 7 = 96 for Iterative Hard Thresholding 
(IHT) [15], 7 = 279 for Subspace Pursuit (SP) [18] and 7 = 424 for Compressed 
Sampling Matching Pursuit (CoSaMP) [19]. 

Not all compressed sensing algorithms achieve the optimal order of k being 
proportional to n. One such algorithm is Orthogonal Matching Pursuit (OMP), 
which has recently been analyzed using RICs, see [30] and references therein. An 
analytic asymptotic sampling theorem for OMP can be deduced from Theorem 
2, see Corollary 7. 

Corollary 7. Given a sensing matrix, A, of size nx N whose entries are drawn 
i.i.d. from K(0, 1/n), in the limit as n/N — > 5 e (0, 1) a sufficient condition 
for recovery for Orthogonal Matching Pursuit ( OMP) is 

n > 2k(k- l)[3 + 21ogAT + logn- 31ogfe]. 

2. Accuracy of main results 

This section discusses the accuracy of Theorems 2-4 and Corollary 5, com- 
paring the expressions with the bounds in Theorem 1, which arc defined [26] 
implicitly in Definition 8. 

Deflnition8. Define £(S, p) and 11(5, p) as 

L(5, p):=l- \ mm (S, p) and U(S, p) := X max (S, p) - 1 (11) 

with H(p) := — plog (p) — (1 — p) log (1 — p) denoting the usual Shannon Entropy 
with base e logarithms, X mm (8,p) and \ max (5,p) as the solution to (12) and 
(13), respectively: 

v& min (A, 6, p) := tfw(A min (5, p),p)+ S^Sp) = (12) 
for X mm (S,p) < 1 - p and 

* max (A, S, p) := i[ max {\ max {5, p),p) + S^RiSp) = (13) 
for X max (5, p) > 1 + p where 

VWi(A, p) := H(p) + ^[(l-p)logA + l-p + pIogp-A], (14) 

Vw(A,p) := ^ [(1 + p) log A + 1 + p - p log p - A] . (15) 
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In Definition 8, the quantities tp m i n (^,p) and ?/Wz(A, p) in (14) and (15), 
are the large deviation exponents of the lower tail probability density function 
of the smallest eigenvalue and the upper tail probability density function of 
the largest eigenvalue of Wishart matrices respectively. The ^min (A, S, p) and 
<P mx (A, 8, p) include a Shannon entropy term from a union bound of the (^) 
submatrices with k columns. The level curve of "JVm (A, 8, p) and * max (A, 8, p) 
defines the transition which for 8 and p fixed it becomes exponentially unlikely 
that the smallest eigenvalue is less that \ mm (8, p) and the largest eigenvalue is 
less than \ max (8,p). 

Theorems 2-4 are discussed in Sections 2.1 - 2.3 respectively. Each section 
includes plots illustrating the formulae and relative difference in the relevant 
regimes. The discussion of Corollary 5 is included in Section 2.3. This Section 
concludes with proofs of the compressed sensing sampling theorems discussed 
in Section 1.1. 

2.1. Theorems 2: 8 fixed and p< 1 

Figure 1, left panel, displays the bounds li(S, p) and £(8, p) from Theorem 1 
for 5 = 0.25, c = 6 and p E (10~ 10 , 1CP 1 ). This is the regime of Theorem 2 and 
the formulae (3) and (3) are also displayed. Formulae (3) and (4) are observed 
to accurately approximate 11(8, p) and £(<5, p) respectively in both an absolute 
and relative scale, in the left and right panel of Figure 1 respectively. 




Figure 1: RIC bounds for 5 = 0.25, c = 6 and p £ (1CT 10 , 10 -1 J. Left panel: 11(5, p), £(<S, p), 
U p (5,p) and L p (8,p). Right panel: relative differences, ^^'^, j tl (^p) anc j l £ (^pj £ (<^p)l ^ 



2.2. Theorems 3: p fixed and 8 <C 1 

Figure 2 displays the bounds U(<5, p) and L(S, p) from Theorem 1 along with 
the formulae (5) and (6) of Theorem 3 in the left and right panels respectively; 
for diversity the upper RIC bound is shown for p — 0.5 and the lower RIC bound 
for p = 0.1, in both instances 8 £ (10 -50 , 10 _1 ) and c = 1. This is the regime 
of p fixed and i « 1 where the upper RIC diverges to infinity and the lower 
RIC converges to its trivial unit bound as 8 approaches zero. The bounds of 
Theorem 3 are observed to accurately approximate 11(8, p) and L(8, p) in both 
an absolute and relative scale, in Figure 2 and 3 respectively. 
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Figure 2: RIC bounds for 5 e (IO- 50 , lO" 1 ) and c = 1. Left panel: 11(8, p) and U s (S,p) for 
p = 0.5. Right panel: C(S,p) and JC s (S,p) for p = 0.1. 




Figure 3: Relative difference in RIC bounds for <5 S (10 50 ,10 1 ) and c = 1. Left panel: 
|U( %>) ( ' ,P) ' for P = °- 5 - m 9 ht Vinel: \^2t^M f or p = .1. 

2.5. Theorems 4: p = (7log(l/<S)) _1 and S < 1 

The left panel of Figure 4 displays the bounds 11(6, p) and £(5, p) from The- 
orem 1 along with the formulae (7) and (8) of Theorem 4 for c u = c\ = 1/3, 7 = 
300 and 6 € (10~ 80 , lO^ 1 ). The formulae of Theorem 4 are observed to accu- 
rately approximate the bounds in Theorem 1 over the entire range of 6; the 
relative differences between these bounds are displayed in the right panel of 
Figure 4. 

The left panel of Figure 4 shows the RIC bounds converging to nonzero con- 
stants as 6 approaches zero, displayed for c u = q = 1/3 and 7 = 300. Corollary 
5 provides formula for 5 <C 1 , which is observed in Figure 5 to accurately approx- 
imate the formulae in Theorem 4 for c u — ci = 1/3 and S = 10~ 80 , uniformly 
over 7 € (1,300). 

2.4. Proof of compressed sensing corollaries 

Corollaries 6 and 7 follow directly from Theorems 4 and 2 and existing 
RIC based recovery guarantees for the associated algorithms in [14] and [30] 
respectively. 
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10"™ 10""° 10"" 10"™ 10"" 10"™ 10""° 10"™ 

5 5 



Figure 4: A comparison of lt 7 (5, p 7 (8)) and -C 7 (5, p 7 (<5)) to U(<5, p) and £(<5, p) respectively 
for c u = c ; = 1/3, 7 = 300 and 8 G (10" 80 , 10" 1 ). Le/t paneZ: UT(5, p 7 (5)), U(<5,p), 
£ 7 ((5,p 7 (<5)), and L 1 {8,p 1 (<5)). Right panel: their relative differences l u ^ ,p ) (£iP2i£12j an< j 

£(*,p) 



\ ---5 KM*)) 

--.^V Figure 5: Plots of U 7 (5,p 7 (5)) and 

^"""^■^^j^ p 7 (<5)) as well as f u (j) and ^(7) 

° ! t l- J given by (9) and (10) respectively, for c u = 

y c ; = 1/3, 8 = lO" 80 and 7 e (1, 300). 

2.4-1. Proof of Corollary 6 

Proof. There is an extensive literature on compressed sensing and sparse ap- 
proximation algorithms which are guaranteed to recover vectors x that satisfy 
bounds of the form ||xo — ^|| 2 < Const ■ 1 1 e )| 2 from y = Axq provided the RICs 
of A are sufficiently small. The article [14] provides a framework by which RIC 
bounds can be inserted into the recovery conditions, and compressed sensing 
sampling theorems can be calculated from the resulting equations. Theorem 4 
establishes valid bounds on the RICs of Gaussian matrices in the regime con- 
sidered in Corollary 6. The claims stated in Corollary 6 follow directly from 
substituting the RIC bounds of Theorem 4 into Theorem 10-13 of [14] and solv- 
ing for the minimum 7 that satisfies the stated theorems; the calculated values of 
7 have been rounded up to the nearest integer for ease of presentation. Nearly 
identical values of 7 can be calculated using the equations from Corollary 5 
rather than the more refined equations in Theorem 4. 

□ 
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2.4-2. Proof of Corollary 7 

Proof. It has been recently shown that Orthogonal Matching Pursuit (OMP) is 
guaranteed to recover any fc-sparse vector from its exact measurements provided, 
[30], 

max(L(k, n, N; A), U(k, n, N; A)) < 1 . (16) 

V k - 1 

The claimed sampling theorem is obtained by substituting the bound from The- 
orem 2 for max(L(fc, n, N; A), U(k, n, N; A)) and solving for n. □ 

3. Proofs of Theorems 2-4 

The proof of Theorems 2-4 are based upon the previous analysis in [22, 26], 
differing in the asymptotic limits considered. The analysis here builds upon the 
following large deviation bounds on the probability of the sparse eigenvalues 
exceeding specified values; these bounds are as follows: 

With L(k,n,N;A) and U(k,n,N;A) defined as in (1) and (2) respectively, 
and f mH (X(5, p),S, p) and (X(5, p), 6, p) defined as in (12) and (13), we 
have the bounds [22, 26] 

Prob ( max X max (A* K A K ) > x) 

\KcQ,\K\=k J 

< poly(n, A) • cxp (2n • * max (A, 5, p)) , (17) 

and 

Prob ( min X min (A* K A K ) > A ) 

\KcQ,\K\=k J 

< poly(n, A) • cxp (2n • * mi „ (A, S, p)) , (18) 

where A mm (i3) and A max (B) are the smallest and largest eigenvalue of B 
respectively and poly(z) is a (possibly different) polynomial function of its ar- 
guments, for explicit formulae see [22]. Theorems 2-4 follow by proving that 
for the claimed bounds, the large deviation exponents n* max (X(S, p),S, p) and 
"■^min (X(S, p), 8, p) diverge to — oo as the problem size increases, and do so at a 
rate sufficiently fast to ensure an overall exponential decay. In addition to estab- 
lishing the claims of Theorems 2-4, we also show that the bounds presented in 
these theorems cannot be improved upon using the inequalities (17) and (18), 
they are in fact sharp leading order asymptotic expansions of the bounds in 
Theorem 1. 

Throughout the proofs of Theorems 2-4 we will be using the following bounds 
for the Shannon entropy function, H(x) := — xlogx — (1 — x) log(l — x) 

H(x) < — xlogx + x, and 

H(x) > -xlogx + x - x 2 ; (19) 
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the upper bound follows from (20) and the lower bound follows from (21), 

-(l-aj)log(l-a;) <x Vxe(0,l), (20) 
- log(l -x)>x Vx < 1 and x ^ 0. (21) 

3.1. Theorem 2 

3.1.1. The upper bound, U p (S,p) 
Proof. Define 



\™x(6,p):=l + ^2p\og^ 7 ^j+cp, => U p (S,p) = \™(S,p)-l 

as from (3). Bounding U P (S, p) from above by (1 + e)U p (5, p) is equivalent to 
bounding from above A™ ax by (1 + e)A™ ax — e. We first establish that for a 

slightly looser bound, with c > 6, the exponent $ maJ[ ^(1 + e)A™ ax — e, S, pj 

is negative, and then verify that when multiplied by n it diverges to — oo as 
n increases. We also show that for a slightly tighter bound, with c < 6, 

f maJ[ ^(1 — e)A™ ax + e, 5, pj is positive, and hence the bound U p (5,p) cannot 

be improved using the inequality (17) from [26]. We show the above properties, 
in two parts that for S fixed: 

1. 3 p , e > & c> 6 such that for p < p , * ma x ((1 + e)A™ ax - e, 6, pj < 0; 

2. $po, e > & c < 6 such that for p < p , * max ((1 - e)A^ nax + e, 6, pj < 0, 
which are proven below separately as Part 1 and Part 2 respectively. 

Part 1: 

2* max ((1 + e) A- ax - e, 8, p) = (1 + p) log ((1 + e)A;° ax - e) 

- plog(p) + p + 1 - ((1 + 6)A- ax - e) + |h(J P ), (22) 

by substituting (1 + e)A™ ax — e for A in (13). We consolidate notation 

using u := A™ ax — 1 and using the first bounds of the Shannon entropy in 
(19) we bound (22) above as follows 



2* max ((l + e)A^ ax -e,<5,pj 

< (l + p)log[(l + e)(l + u) -e] - p\ogp + p+ 1 - (1 + e)(l + u) 
+ e+- 5 [-8p\og{5p) + 5p], (23) 

= (1 + p) log [1 + (1 + e)u] + plog (j^j +p-u-6u + 25p. (24) 
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From (23) to (24) we expanded the products of (l+e)(l+u) and simplified. 

Now replacing plog (^jr^^j by its equivalent (u 2 — cp) /2 and expanding 
(1 + p) in the first term we bound (24) by 

2* max ((l + e)A- ax -e,(5,p) 



< log (1 + u + eu) + p log (1 + u + eu) + ^ (u 2 — cp) + 3p — u — eu, 



(25) 



£!l \ 1 2 f 

log(l + u) + log 1 + — — + -u - -cp + 3p~u- eu 
1 + u I 2 2 



+ (0 log(l + U ) + plog^l + ^-j , (26) 

1 o 1 3 £W 1 2 1 / r.\ 

<«--« + 5 « +5^ + 2" --(c-ev-u-™ 
+ „ + £L. (27) 

From (25) to (26) the term log(l + it + eu) is factored as in the first two 
logarithms in (26). From (26) to (27) we bounded the first log(l + u) 
from above using the second bound in (28) and bounded above all other 
logarithmic terms using the first bound in (28) . 



log(l + x) < x, (28) 
log(l + a;) < x-^-x 2 + -x 3 Vx > -1. 

We can bound above 1/(1 + u) in the fourth and last terms of (27) using 
the bound of (29) below. 

< 1 for < x < 1. (29) 



1 + x 

Therefore, (27) becomes 



2* max ((l + e)A™ ax -e, ( 5,p) 



< ^-u 3 — i (c — 6)p — eu + eu + pu + epu, (30) 
= -\{c-Q)p+\u 3 + {l + e)pu, (31) 

< ~\(c ~ 6)p - \{c - 6)p + \u 3 + 1(1 + e)u 3 , (32) 
= ~(c -Q)p-\{c- 6)p + ^llu 3 - (33) 
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We simplified (30) to get (31). From (31) to (32) we split the first term into 
half and bounded above pu by u 2 /14 using the fact that by the definition 
of u, 



u 2 = p 



2 log 



1 



S 2 p s 



+ 7 



1 



9 1 9 

u < p< — u . 
14 



Then we simplified from (32) to (33). 

Now in (33), if the sum of the last two terms is non-positive there would 
be a unique p such that as p — > for any p < po and fixed 5 (33) will be 
negative. This is achieved if c > 6 and 



-i(c-6)p + il±^ 3 <0 



, 21(c-6) 



2(17 + 3e)' 



(34) 



Since u is strictly decreasing in p, there is a unique po that satisfies (34) 
and makes (33) negative for 5 fixed, e > 0, c > 6 and p < p as p — > 0. 

Having established a negative bound from above and the po for which it is 
valid, it remains to show that n ■ 2* max ^(1 + e)A™ ax — e, 5, p^ -4- — oo as 

(k,n,N) — > oo. The claimed exponential decay with fc follows by noting 
that n • p = k, which in conjunction with the first term in the right hand 
side of (33) gives a concluding bound — (c — 6)fc/4. For p < po therefore 



Prob (U(k, n, N; A) > (1 + e)U p (8, p)) 

< po/y ^n, (1 + e)A 



max 
P 



• exp 



(c-6)fc 



The above bound goes to zero as /c — > oo provided (log n)/k — > so that 
the exponential decay in /c dominates the polynomial decrease in n. 



Part 2: 

2* max ((1 - 6)A^ ax + 6, 5, p) = (1 + p) log ((1 - e)A- ax + e) 

- plog(p) + p + 1 - ((1 - e )A- ax + e) + - S R(S P ), (35) 

by substituting (1 — e)A™ ax + e for A in (13). We consolidate notation 
using u := A™ ax — 1 and bound the Shannon entropy function from below 
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using the second bound in (19) to give 
2* max ((l-e)A™ ax + e,<5,p) 

> (1 + p) log [(1 - e)(l + u) + e] - plogp + p + 1 - (1 - e)(l + u) 

- e + | [Sp log (Sp) + Sp- S 2 p 2 ] , (36) 

= (1 + p) log [1 + (1 - e)u] + plog (j^j + 3p - (1 - e)u - 25 p 2 . (37) 

From (36) to (37) we expanded the products of (1 — e)(l+u) and simplified. 

Now replacing plog (^jr^sj by (u 2 — cp) /2 and expanding (1 + p) in the 
first term we have (37) become 



((l-e)A-- + e,5,p) 
H 

(1 - e)u - 25p 2 , (38) 



> log [1 + (1 - e)u] + plog [1 + (1 - e)u] + ^ (w 2 - cp) + 3p 



> (1 - e)u - - ^ u 2 + i?i 2 - icp + 3p - (1 - e)u + p(l - e)u 



2 



p?i 2 - 2<5p , (39) 



u + - (6 - c)p + pu - epu — pu - 25 p , (40) 

> l(Q- c )p+^pu-26p 2 . (41) 

From (38) to (39) we bounded below the logarithmic terms by the first 
two terms of their series expansion using (42) 

log(l + x) > x - \<c 2 Vx>-1. (42) 

From (39) to (40) we bounded above pu 2 and (1 — e) 2 by pu and 1 — e 
respectively and simplified. Then we dropped the first term to bound 
below (40) by (41) and we simplified the terms with pu. 

For c < 6, the only negative term in (41), the last term, goes faster to 
zero than the rest. Therefore, there does not exist a p , e > and c < 6 
such that for p < po and fixed 5 (41) is negative. Thus the bound 

Prob (U(k, n, N; A) > (1 - e)U p (5, p)) 

< poly (n, (1 - e)A™ ax + e) • exp [2n* max ((1 - e)A; nax + e, 6, p) 
does not decay to zero as n — > oo. 
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Now Part 1 and Part 2 put together shows that U P (S, p) is a tight upper 
bound of U(k, n, N; A) with overwhelming probability as the problem size grows 
in the regime prescribed for VL P (5, p) in Theorem 2. 

□ 

3.1.2. The lower bound, LP (5, p) 
Proof. Define 



Xf-(S,p):=l- ] j2p\og^^j+cp, => Z"(6,p) = l-\f n (S,p) 

as from (4). Since bounding L p (S,p) above by (1 + e)L p {8 1 p) is equivalent to 
bounding A™ ln above by (1 + e)A™ m — e. We first establish that for a slightly 

looser bound, with c > 6, the exponent "fmin ^(1 + e)A™ m — e,5,p*J, and then 
verify that when multiplied by n it diverges to — oo as n increases. We also 
show that for a slightly tighter bound, with c < 6, ^min ((1 — e )^™ m + e i 8, p^j is 

positive, and hence the bound £ p (5, p) cannot be improved using the inequality 
(18) from [26]. We show, in two parts that for 6 fixed: 



1. 3 po, e > & c> 6 such that for p < p , * mi „ (Jl + e)A™ 111 - e, 5, pj < 0; 

2. $ po, e > & c < 6 such that for p < p , * m i„ ((1 - e)A™ in + e, S, p) < 0, 
which are proven separately in the two parts as follows. 

Part 1: 



((1 + e)Af n - e, (5, p) = 2H(p) + (1 - p) log ((1 + e)Af n - e) 

+ plog(p) - p + 1 - ((1 + e)Af n - e) + H H (Jp), (43) 



by substituting (l + e)A" lm — e for A in (12). We consolidate notation using 

I := 1 — A™ m and bound the Shannon entropy functions from above using 
the first bound in (19) which gives 

2* min ((l + e)Af n -e, ( 5,p) 

< -2p log (p) + 2p + (1 - p) log [(1 + e)(l - I) - e] + p log p 

- p + 1 - (1 + e)(l - /) + e - 2plog(«5p) + l[Sp), (44) 

= (1 - p) log (1 - I - el) + plog f ] + 3p + l + el. (45) 



6 2 p 



We simplified from (44) to (45). 
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Now replacing plog ^^3^ by (l 2 — cp) /2 and factoring (1 — I) in the 
argument of the first log term we have (45) become 

2* mi „((l + £)Af n -e,(5,p) 

< (1 - p) log(l - + (1 - p) log (1 - + 1 (l 2 - cp) + Zp 

+ l + d, (46) 

< / + log(l -l) + ^l 2 - ^cp + 3p- plog(l - I) + d 

-(i-p)i4t' ( 4? ) 

< I - I - \l 2 + \l 2 - \{c - 6)p - plog(l -l) + d- d(l - p), (48) 

= ~{c - 6)p - plog(l - + eZ - eZ + ep/, (49) 

= -\{c- 6)p - i(c - 6)p - plog(l - + epl. (50) 

From (46) to (47) we expanded (1 — p) and we bounded above the second 
logarithmic term using the first bound of (51). 

log(l-a;) < -x, (51) 
log(l -x) < -x- ^x 2 , 

log(l-a;) < -x-^x 2 -^x 3 Va;e(0,l). 

From (47) to (48) we bounded above the first logarithmic term using the 
second bound of (51) and also bounded 1/(1 — I) using (52). 

' >1 Vze(0,l). (52) 



l-x 



From (48) to (49) we expanded the last brackets and simplified and from 
(49) to (50) we simplified and split the first term into two equal terms. 

Equation (50) is clearly negative if c > 6 and the sum of the last three 
terms is non-positive, which is satisfied if el — log(l — I) < (c — 6)/4, which 
is also true if, using the first bound in (28), (1 + e)l < (c — 6)/4. Since 
I is strictly increasing in p, taking on values between zero and 1, there is 
a unique po such that for fixed S, e > and c > 6, any p < po satisfies 
(1 + e)l < (c- 6)/4 and (50) is negative. 

Having established a negative bound from above and the p for which it is 
valid, it remains to show that n ■ 2vE f min ^(1 + e)A™ m — e, 5, p^j — > —00 as 
(k,n,N) -4- 00, which verifies an exponential decay to zero of the bound 
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(18) with k. This follows by noting that n ■ p — k, which in conjunction 
with the first term in the right hand side of (50) gives a concluding bound 
— |(c — Q)k. For p < po therefore 

Prob (L(k, n, N; A) > (1 + e)Z p (6, p)) 

< poly (n, (1 + e)Af n - e) • exp 

The right hand side of which goes to zero as k — > oo with (logn)/fc — > 
as k — > oo so that the exponential decay in k dominates the polynomial 
decrease in n. 



(c-6)fc 
4 



Part 2: 

2* mi n ((1 - e)\f n + e, 5, p) = 2H(p) + (1 - p) log ((1 - e)\^ + e) 

+ plog(p) - p + 1 - ((1 - e)Af 11 + e) + |h(J P ), (53) 

by substituting (1 — e)A" lln + e for A in (12). We consolidate notation using 
I := 1 — A™ 111 and bound the Shannon entropy function from below using 
the second bound in (19) to give 

2* min ((l-e)Af n + e,5,p) 

> 2 [-plogp + p - p 2 ] + (1 - p) log [(1 - e)(l - + e] + plogp - p 
+ 1 - (1 - e)(l - - e + - & [-plog (Sp) + 5p- 5 2 p 2 } , (54) 

= -2plogp + 2p- 2p 2 + (1 -p) log [1 -e - (1 -e)l + e] +plogp-p 
+ 1 - 1 + e + (1 - e)l - e - 2plog {Sp) + 2p- 2Sp 2 , (55) 

= log [1 - (1 - e)l] + (1 - e)l - plog [1 - (1 - e)l] + plog ^-L) 

+ 3p-2(l + (5)p 2 . (56) 

From (54) to (55) we expanded brackets and simplified and further sim- 
plified from (55) to (56). 

Now replacing plog^j^^ by (l 2 — cp) /2, bounding above the second 
logarithmic term using the first bound of (51) and factoring out log(l — I) 
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we have 
2tf min ((l-e)Af n + e,<5,p) 

> log(l - /) + log (\ + ^-j^j +l-d + (l- e)pl + \{l 2 - cp) + 3p 

- 2(1 + 5)p 2 , (57) 

> log (1 - I) + I + i/ 2 - ^cp + 3p-el + log (1 + el) + pi - epl 

- 2(1 + 8)p\ (58) 
>-l- ]f - \f + I + \l 2 + \{& - c)p + pl-el + el- |e 2 Z 2 - epl 

-2{l + 5)p\ (59) 

= 5(6 - c)p \l 3 + pl- 2(1 + 5)p 2 l -eH 2 - epl. (60) 

From (57) to (58) we bounded below 1/(1 — I) using (52). From (58) to 
(59) we bounded below the first logarithmic term using 

log(l - x) > -x - -x 2 - -x 3 Vx e [0,0.44], (61) 

and also bounded below the second logarithmic term using (42). From 
(59) to (60) we simplified. 

The dominant terms in (60) are the first two term, all the rest go to zero 
faster as p — > 0. Therefore, for (60) to be positive as p — > we need the 
sum of the first two terms to be positive. This means 

i(6-c)p-lj 3 >0 => l 3 <(6-c)p. (62) 

This holds for c < 6 and small enough p and since I is a decreasing 
function of p^ 1 there would not a p below which this ceases to hold 
as p — > 0. Hence we conclude that for fixed S, e > and c < 6 
there does not exist a po such that for p < po, (60) is negative and 

2*min ((1 - e)A^ in + e,5,pj < as p -> 0. Thus 

Prob (L(k, n, N; A) > (1 - e)Z p (6, /?)) 

< poly (n, (1 - e)Af n + e) • exp [2n* min ((1 - e)A« lin + e, 6, p) 

and as n — > oo the right hand side of this does not go to zero. 

Now Part 1 and Part 2 put together shows that L p (S,p) is also a tight 
bound of L(k, n, N; A) with overwhelming probability as the problem size grows 
in the regime prescribed for L p (S,p) in Theorem 2. 

□ 
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3.2. Theorem 3 

3.2.1. The upper bound, U 5 {S,p) 
Proof. Define 

\r x (5,p) :=l + 3p + plog 



( 1 



(1 + p) log 



clog 



5 2 p 3 



It follows from (5) that U 5 (S,p) = Xf^ x (8,p) - 1. Bounding U 5 (5,p) above by 
II s (5, p) + eis equivalent to bounding A™ ax above by A™ ax + e. We first establish 



that for a slightly looser bound, with c > 1, the exponent ^ max ^A™ ax + e, 5, p 

is negative and then verify that when multiplied by n it diverges to — oo as 
n increases. We also show that for a slightly tighter bound, with c < p, the 



exponent 



A" 



e,S,p) is bounded from below by zero, and hence the 



bound U s (5, p) cannot be improved using the inequality (17) from [26] We show, 
in two parts that for p fixed: 

1. 3 5 , e > and c> 1 such that for 5 < 5 Q , * mal (A^ ax + e, 8, pj < 0; 



~, 8, pj 



< 0. 



2. $ 5 , e > and c < p such that for S < 5 , * max ^A^ ax 
which are proven separately in the two parts as follows. 
Part 1: 

2* ma * (A> ax +e,S,p)=(l + p) log (A> ax + e) 

- plog(p) +p+l- (\r x + e)+ - s H(Sp), (63) 

by substituting A™ ax + e for A in (13). We bound the Shannon entropy 
function above using the first bound of (19) and consolidate notation using 
1, then (63) becomes 



u := A™ ax 



2* max (Ar x + M,p) 

< (1 +p)log [(1 + it) + e] - p\ogp + p+ 1 - (1 +u) - e 



+ -[-Splog(5p) + Sp], 
■ (1 + p) log (1 + u + e) + plog 



d 2 p 3 



(64) 

+ 3p — u — e. (65) 



From (64) to (65) we simplified. Next where u is not in the logarithmic 
term we replace it by p log (yjr^j + (1 + p) log c log {^jr^j + 3p to have 
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2* 



< (1 + p) log (1 + u + e) + plog 
/ 1 



5 2 p ? 



3/0- p log 



3p 



- (1 + p) log 



clog 



= (1 + p) log (1 + u + e) - e - (1 + p) log 



clog 



1 



= -a(l + p) -e+(l + p)lo. 
= -a - ap - e + (1 + p) log 
< -a + (1 + p) log 



1 + u + e 
cl °g(p7 
1 + u + e 



e a (l + u + e) 
cl °g(^) . 



(5 2 p 3 

+ a(l + p), 
■a(l + /9) loge, 



(66) 
(67) 

(68) 
(69) 
(70) 



From (66) to (67) we simplified and from (67) to (68) we combined the 
logarithmic terms and to create a constant we add — a(l + p) and a(l + p) 
for a small positive constant < a < 1. From (68) to (69) we rewrote 
a(l + p) as a(l + p)\oge. From (69) to (70) incorporated the second 
logarithmic term into the first one and we bounded above (69) by dropping 
the — e and — ap. 

Equation (70) is clearly negative if the second term is negative, which is 
satisfied if the argument of the logarithm to be less than one. This leads 
to 

1 



*clog 



>u + l + e, 



where again substituting p\og (^jr^j + (1 + p) l°gl°g ( y s^- 
and reordering the right hand side of (71) gives 



(71) 
3p for u 



*clog 



<5V 



> log log 

+ p 



1 



S 2 p 3 

3 + log 



+ l + e 
1 



S 2 p s 



+ log log 



S 2 p s 



(72) 



For small < a < 1 and c > 1, the left hand side of (72) is an unbounded 
strictly increasing function of S^ 1 growing exponentially faster than the 
right hand side of (72). Consequently there is a unique So for which the 
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inequality (72) holds for fixed p, e > 0, c > 1 and any S < S and as a 
result making 2* max ^A|f ax + e,S,pj < 0. 

Having established a negative bound from above and the 8q for which it 

is valid, it remains to show that n ■ 2* max ^A™ ax + e, 5, pj — > — oo as 

(k,n,N) — » oo, which verifies an exponential decay to zero of the bound 
(17) with n. This follows from the first term of the right hand side of (70), 
giving a concluding bound n(— a). For 5 < S therefore 

Prob (ll(k, n, N; A) > U S (S, p) + e) 

< poly (n, A£ lax + • cxp (-an) . 
The right hand side of which goes to zero as n — > oo. 



Part 2: 



2* max (\T X - C P) = (1 + P) log (A^ ax - e) 



- plog(p) +p+l- (Ar x - e) + -H(Jp), (73) 

by substituting A™ ax — e for A in (13). We lower bound the Shannon 
entropy function using the second bound of (19) and consolidate notation 



:=AS 



using u .— ,\ s 

(Xr* e,s, P 



— 1, then (73) becomes 



> (1 + p) log[(l +u) - e] - p\ogp + p+ 1 - (1 + u) +e 
2 
6 

/ i \ 

3p — u + e — 2(5p 2 , 



2plog (5p) + - [-Jplog (Sp) +Sp- 8 2 p 2 ] , 

1 



= (1 + p) log (u + 1 - e) + p log 



(1 + p)log(u + 1 - e) +plog ( -J-j ) + 3p- plog 



(1 + p) log 



clog 



1 



<5V 



e-2(5p 2 , 



<5 2 p J 



3 P 



(74) 
(75) 

(76) 



(1 + p) log (u + 1 - e) - (1 + p) log 
1 + u - e 



clog 



<5V 



e + (1 + p) log 



_clog(^)_ 



2«5p 2 . 



+ e-2<5p 2 , (77) 



(78) 



From (74) to (75) we simplified. Then from (75) to (76) we replace u by 
plog (yjr^j +(l+p) log clog (^jr^j +3/0 where u is not in the logarithmic 
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term. From (76) to (77) we simplified and from (77) to (78) we combined 
the logarithmic terms. 

The last term in (78) obviously goes to zero as S — »■ 0, then for the expres- 
sion to remain positive we need to know how the dominant term, which 
is the second term, behaves. For this term to be nonnegative as S — > 
for fixed p we need the argument of the logarithmic to be greater than or 
equal to 1 which means the following. 



u + 1 + e > c log 



Therefore substituting for u we have 



1 



S 2 p s 



p\og 



1 



S 2 p 3 



+ (l + p) log 



clog 



5 2 p 3 



+ 3/?+ l + e> clog 



S 2 p s 



Then we expand the second logarithmic term and rearrange to get 



[p - c) log 



1 



S 2 p 3 



+ (1 + p) log 



clog 



1 



6 2 p 3 



+ 3p+l + e >0. (79) 



Inequality (79) is always true for fixed p and c < p as S — > 0. Therefore, 
we conclude that there does not exists 6q such that for any p fixed and 
e > for S < 5 ( 78) is negative and 2* max ^A^ ax - e,S,pj < as S -> 0. 
Thus 



Prob (U{k, n, N; A) > II s (6, p) - e) 



< 



poly (n, A£ lax - e) ■ exp [2n* max (A^ ax - e, <5, p) 



and as n — > oo the right hand side of this does not necessarily go to zero. 

Now Part 1 and Part 2 put together shows that U s (5, p) is also a tight 
upper bound of U (k, n, N; A) with overwhelming probability as the problem 
size grows in the regime prescribed for VL S (5, p) in Theorem 3. 

□ 



3.2.2. The lower bound, L s {S,p) 
Proof. Define 



Af"(M :=cxp 



3p + c 
1-P 



(6 2 p 3 ) — 



L s (5,p) = l 



\ mm 

A <5 



as from (6). Bounding £ s (5, p) above by (l + e)L s (S, p) is equivalent to bounding 
A™ 111 above by (1 + e)A™ m — e. We first establish for a slightly looser bound, with 

c > 1, the exponent ^min ((1 + e)A™ ln — e, S, p^j is negative and then verify that 
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when multiplied by n it diverges to — oo as n increases. We also show that for a 
slightly tighter bound, with c < 1, the exponent \I/ m i n ^(1 — e)A™ m + e,S, pj is 

bounded from below by zero, and hence the bound L S (S, p) cannot be improved 
using the inequality (18) from [26]. We show, in two parts that for p fixed: 

1. 3 5 , e > and c > 1 such that for S < 5 , * min ((1 + e)A^ in - e, 5, p^j < 0; 

2. $S , e > and c < 1 such that for 5 < 5 , * min ((1 - e)Xf in + e, 5, pj < 0, 
which are proven separately in the two parts as follows. 

Part 1: 

2* mi „ ((1 + e)Af n - C *, P ) = 2H(p) + (1 - p) log ((1 + e)Xf ~ 

+ plog(p) - p + 1 - ((1 + e)Af n - e) + H H (J P ), (80) 

by substituting (1 + e)A™ ln — e for A in (12). We now upper bound the 
Shannon entropy terms using the first bound of (19) and factor out A™ ln 
for (80) to become 

2* min ((l + e)Af n -e, ( 5,p) 

< 2 [-plogp + p - p 2 ] + (1 - p) log (Af n ) - (1 + e)Af 1 + e + l- P 
"(l + e)Af n ~ 



+ plogp+(l~ p) log 



\ mm 
A 8 



+ l[-p\og{8p) + 5p], (81) 



(1 - p) log (Af ") - (1 + e)Xf n + e + (1 - p) log 



(1 + e) 



\ min 

A <5 



+ Pl0g (^) +3P+1 - (82) 

From (81) to (82) we simplified. Using the fact that by the definition of 
Z s (S,p) in (6) 

los( ; r) = -^_ log (_L)-^, 



23 



we substitute this in (82) for log ( A™ 111 I to get 



2* 



((l + e)A 



min 

«5 



<(i-p) 



■log 



€,S,p 
( 1 



3/9 + c 



+ (l-p)log 
+ (1 - p) log 



1 — p \<5 2 p 3 / 1 — p 



(l + e)-~ 



A mm 
<5 



+ plog 



3p-c - (l + e)Ar i + e + plog 



-(l + 6)Af n + e 

' ' f:V+l. 
1 



(83) 



S 2 P S 



(1 + e) 



\ min 
A 5 



= (1 - p) log 



+ 



\ mm 

A <5 



+ 3p+l, 

Af n -eAf n -(c-l) + e. 



(84) 
(85) 



From (83) to (84) we expanded the brackets and from (84) to (85) we sim- 
plified. Now we consolidate notation using I := 1 — A™ 111 and substituting 
this in (85) we have 



2* r 



((l + e)Af n -M,p) 



< (1 - p) log 



(1 + e)- 



l-l 



= -(c-l) + (l-p)log (^1- T 

<-(c-i)+d-(i-p) I 4 T -(i-o 

= -I(c-l)-I( c -l) + e i. 



-(l-/)-e(l-0-(c-l) + e, (86) 



(1 - /) + eZ, 



(87) 
(88) 
(89) 



From (86) to (87) we simplified and from (87) to (88) we bounded above 
the logarithmic term using the first bound of (51). From (88) to (89) 
we dropped the third and fourth terms, which are negative, and split 
the leading term into half. Inequality (89) can be further bounded by 
— (c — l)/2 (which will be negative if c > 1) by choosing e to be less than 
(c - l)/2 and noting that I € (0, 1]. 

Having established a negative bound from above and the So for which it is 

valid, it remains to show that n ■ 2\I/ m i n ^(1 + e)A™ ln — e, 5, p^j — > — oo as 

(k,n,N) — > oo, which verifies an exponential decay to zero of the bound 
(18) with n. This follows from the first term of the right hand side of (89) 
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(c — l)n 



giving a concluding bound — ^(c — l)n. For S < So therefore 

Prob (L(k, n, N; A) > (1 + e)Z s (8, p)j 

< poly (n, (1 + e)Af n - e) • exp 
The right hand side of which goes to zero as n — > oo. 

Part 2: 

2* mi „ ((1 - e)Af n + e, 5, p) = 2H(p) + (1 - p) log ((1 - e)Af n + e) 

+ plog(p) - p + 1 - ((1 - e)Af n + e) + - s K(6p), (90) 

by substituting (1 — e)A™ m + e for A in (12). Next we bound the Shannon 
entropy functions from below using the second bound in (19) to give 

2* mi „((l-e)Af n + e,(5,p) 

> 2 [-plogp + p p 2 ] + (1 - p) log (xf n ) - (1 - e )Af n + 

"(1 -e)Af n + e" 



+ plogp -p + (1 -p) log 



\ min 



+ - s [-p\og(5p)+Sp-S 2 p 2 }, (91) 



(1 - p) log (Af n ) - (1 - e)Af n - e + (1 - p) log 



\ mm 
A <5 



+ plog (s^) +3p+l-2(l + % 2 . (92) 

From (91) to (92) we simplified. Using the fact that by the definition of 
Z s (5,p) in (6) 



1-P 



we substitute this in (92) for log ^A|f m ^ to get 
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2* mi „((l-e)Af n + e, ( 5, /9 ) 



>(1-P) 



P 



1-P 



log 



S 2 p s 



3p 



+ (1 - p) log 
= -plog 



\ min 

A <5 



1-P 

f plog 



(l-6)Ar n -e + 3p 



/ 1 



v c5 2 p 3 
+ (1 - p) log 



3p-c-(l-e)A? 



V<5 2 p 3 
- e + plog 



l-2(l + <5)p 2 , (93) 
/ 1 



(1-0 + 



\ min 
A <5 J 



= (1 - P) log 



(1-0 + 



\ min 

A s 



+ 3p+l-2(l + <5)p 2 , 



\ min i \ min 
A <5 + eA <5 



2(l + 5)p 2 . 



(94) 



(95) 



From (93) to (94) we expanded the brackets and from (94) to (95) we sim- 
plified. Now we consolidate notation using I := 1 — A mm and substituting 
this in (95) we have 



2*mi„ ((1 

> (1 - P) log 



e)Ar n + e 



><*. p) 



(1-0 + 



1-Z 



(l-Z) + e(l-Z)-e + l 



-2(l + % 2 , 
(l-p)logfl 



_eZ_ 

I - 7 



Z - c-eZ- 2(1 + 6)p 2 , 



> (1 - p) log (1 + el) + I - c - el - 2(1 + <5)p 2 , 

> (1 - p) [el - \t 2 l 2 ^j + I - c - el - 2(1 + (5)p 2 , 

= eZ - ie 2 Z 2 - epl + i £ 2 pZ 2 + Z - c - eZ - 2p 2 - 2(5p 2 , 
= l-c-2p 2 -el-epl- l -e 2 l 2 + l -e 2 pi 2 - 28 p 2 . 



(96) 
(97) 
(98) 
(99) 

(100) 
(101) 



We simplified from (96) to (97) and from (97) to (98) we bounded below 
1/(1 - Z) using the bound of (52). From (98) to (99) we bounded below 
the logarithmic term using the bound of (42). From (99) to (100) we 
expanded the brackets and from (100) to (101) we simplified. 

The leading terms of (101) are the first three and Z is strictly increasing as 
<5 _1 approaches 1. If c < 1, there will be some values of p for which (101) 
will always be positive as S — > 0. Thus there does not exist any So such 
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that for any p fixed, e > 0, c < 1 and 8 < So, (101) becomes negative. 
Thus 

Prob (L(k, n, N; A) > (1 - e)Z 5 (S, p)) 

< poly (n, (1 - e)Af n + e) • exp [2n* min ((1 - e)\f n + e, 5, p)] , 

and as n — > oo the right hand side of this does not necessarily go to zero. 

Now Part 1 and Part 2 put together shows that L 5 (5,p) is also a tight bound 
of L(k, n, N; A) with overwhelming probability as the sample size grows in the 
regime prescribed for L s (S,p) in Theorem 3. 

□ 

3.3. Theorem 4 

3.3.1. The upper bound, U 1 (S, p-y(S)) 

Proof. To simplify notation we will use p for p 7 (<5) in the proof. Lets define 



A-(,>-. / M:=t-,/2 / ,!og(^) + (v> 



It follows from (7) that W(S,p) = A™ ax (<5,p) - 1. Bounding W(S,p) above by 
IP (S, p) + e is equivalent to bounding A™ ax above by A™ ax + e. We first establish 
that for a slightly looser bound, with c u > 1/3, the exponent $ max ^A™ ax + e, S, pj 
is negative and then verify that when multiplied by n it diverges to — oo as n 
increases. We also show that for a slightly tighter bound, with c u < 1/5, the 

exponent ^ maK ^A™ ax — e, S, pj is bounded from below by zero, and hence the 

bound IP (8, p) cannot be improved using the inequality (17) from [26]. We 
show, in two parts that for 7 > 70 fixed: 

1. 3 6 , e > and c u > 1/3 such that for S < S , *,„a X (A™ ax + e, S, pj < 0; 

2. $ So, e > and c u < 1/5 such that for S < S , *,„ax (A™ ax - e, S, pj < 0. 
which are proven separately in the two parts. 



Part 1: 



2* max (A- ax + e, S, p)=(l + p) log (A- ax + e) - plog (p) 

2 
S 



+ P+1- A™ ax - e + -H (Sp) , (102) 



by substituting A™ ax + e for A in the definition of ^ max (A, S, p) in (13). 
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Now letting u = A™ ax — 1 and substituting this in (102) and upper bound- 
ing the Shannon entropy term using the first bound of (19) gives (103) 
below 

2* max (A™ ax + e,5,p 

< (1 + p) log (1 + u + e) - plog (p) + p + 1 - (1 + u) - e 

+ - 8 [-5p\og{5p) + 5p), (103) 

= log(l + u + e) + plog(l + u + e) - u - e + plog (j^j + 3p, (104) 
= log(l + u) + log M + j-^— J + p log(l + u + e) - u - e 



+plog ( v w +3p ' (105) 

< -u + u - ^u 2 + i?i 3 + plog (^^j + 3p + plog(l + u + e) - e 

+ log(l + e), (106) 

< -^m 2 + t;u 3 + plog f ^ ) + 3p + plog(l + w, + e) -e + e. (107) 



2 3 ' b \S 2 p 

From (103) to (104) we expanded the (1 + p) in the first term and sim- 
plified while from (104) to (105) we expanded the first logarithmic term. 
From (105) to (106) we bounded above log(l + u) and 1/(1 + u) using 
the second bound of (28) and the bound of (29) respectively. Then from 
(106) to (107) we simplified and bounded above log(l + e) using the first 
bound of (28). 

Let x — 2plog {^jTpS^j + 6p which means u = yfx + c u x. We simplify (107) 

and replace the sum of the second two terms by \x and u in the first two 
terms by y/x + c u x to get 



2* 



max 



(\™ x + e,5, p 



<~{Vx + c u x) 2 + i (Vi + c u xf + i.x + plog(l + u + e), (108) 
+ plog(l + u + e), (109) 

+ plog(l + w + e). (110) 

From (108) to (109) we expanded the first two brackets and from (109) to 
(110) we simplified. Substituting 1/ [7 log (|)] for p in the expression for 
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x we have x = 4/7 + g(p) where g(p) = 6plog (1/p) + 6p and goes to zero 
with S. Therefore, if 4/7 < 1 for 5 small enough we will have x < 1. This 
means for 7 > 4 we can define Si such that for S < Si, x < 1 and we can 
upper bound a; 5 / 2 and a; 3 by x 2 since x 2 > x 2+ i for j > when a; < 1. 
Using this fact we can bound (110) above to get 



2* m a X (A7 x + e,<5,p) 



( I ■ ) — J! — — „' ._ ? Z 

^ ~ [ c u — — 1 1 c M a; — T^ c u x 1 c u x 1 q c u x 

•plog(l + « + e), (111) 

(/ ^ 3 2 1 (r - \ /■ - • r ■ • !,- :i r- 



= -2^-3; x - H Cu " 3 ) x + CuX + r> + r» x 

+ plog(l + u + e). (112) 

From (111) to (112) we simplified and split the first term into half. The 
last term goes to zero with 5 so we can define 82 such that for S < S2 we can 
bound this term above by x 2 . But also x 3 / 2 = 8/ \/j^+G(p) where G(p) is 
the difference between [4/7 + g(p)] 3 ^ 2 and (4/7) 3 ^ 2 which also goes to zero 
with S because this difference is a sum of products with g{p). This means 
_ x 3/2 < -8/y^3 s i nce5 ( / ri) is positive. Now let f u (c u ) = c„ + c 2 /2 + c 3 /3, 
which is positive for all c u > 0, using the above therefore we can bound 
(112) to get 

2^ max (\^ + e,S,p 

< \ (c - i) • [~j= J - i (c„ - i ) x 3 / 2 + f u (c u )x 2 + x 2 , (113) 
' f ^-^-\(c u - l -)x^ 2 + [l + f u {c u )]x 2 . (1 14) 




7 3 V 37 2 V 3, 

From (113) to (114) we simplified. For (114) to be negative all we need is 
for c u > 1/3 and the sum of the last two terms to be non positive, that is: 

-i(^-i)^ + [ i + /. M ^ S o • ^^__L_} 2 . 

(115) 

Let's define $3 such that for 5 < S3 (115) holds; since a; is a decreasing func- 
tion of S^ 1 for fixed 7 there exist a unique S3. We set So = min (61,82, 83) 
and conclude that if c u > 1/3, for fixed 7 > 70 = 4 and e > when S < So 
as 5 -> (114) will remain negative and 2\f , max ^A™ ax + e, <5, < 0. 

Having established a negative bound from above and the i5o for which it 
is valid, it remains to show that n ■ 2* max ^A™ ax + e, S, p^j — > —00 as 
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(k,n,N) — > oo, which verifies an exponential decay to zero of the bound 
(17) with n. This follows from the first term of the right hand side of 
(114), giving a concluding bound — n ■ 4 (c„ — 1/3) / V7 3 - For fixed 7 > 70 
and S < So therefore 

Prob (ll(k, n, N; A) > VP (6, p) + e) 

<poly (n,A™ ax + e) • exp 
The right hand side of which goes to zero as n — > 00. 



An ( I 



7 3 V 3 



Part 2: 

2* max (a™ x - e, (5, p) = (1 + p) log (a™ x - e) - plog (p) 

2 



p + 1 - A™ ax + e + -H (Sp) , (116) 



by substituting A™ ax — e for A in the definition of \l/ max (A, <5, p) in (13). 

Now letting u = A™ ax — 1 and substituting this in (116) and lower bounding 
the Shannon entropy term using the second bound of (19) gives (117) 
below 



2* max (A™ ax -e,<5,p) 



> (1 + p) log (1 + u - e) - plog (p) + p + 1 - (1 + it) + e 
+ l[-Splog(Sp)+Sp-6 2 p 2 ], (117) 

= log(l + u - e) + p log(l + u-e)-it + e + plog( -J-^ J + 3p 



-25p\ (118) 

= log(l + u) + log ( 1 - — - — ) + p log(l + u — e) - u + e 
V 1 + w/ 

+ plog (s^) +3p-2Sp 2 , (119) 
> -u + u- i u 2 + i u 3 + plog (^^3) +3p + e + log(l-e) 

+ plog(l + it-e)-2(5p 2 , (120) 
= -iw 2 + i« 3 + plog (j^j +3p + e + log(l -e)+plog(l + u-e) 

-25p 2 . (121) 

From (117) to (118) we expanded the (1 + p) in the first term and sim- 
plified while from (118) to (119) we expanded the first logarithmic term. 
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From (119) to (120) we bounded above 1/(1 + u) using the bound of (29) 
and bounded below log(l + u) using the following bound. 

log(l + x) > x - \x 2 + lx 3 Vx e [0,0.921. (122) 
2 5 



From (120) to (121) we simplified. Let x = 2plog (^35^3 J +6/0 which means 

u = \fx + c u x. We simplify (121) and replace the second two terms by 
x/2 and u in the first two terms by y/x + c u x to get 



2* max (A™ ax -e, ( 5,p) 

2 (y/x~ + C u x) 2 + 



> -\ {Vx + c u x) 2 + i (s/x + c u x) 3 + \-x + e + log(l - e) 



+ /9log(l + M-e)-2(5p 2 , (123) 

1 3/2 ^ 2 2 ~^ 3/2 ^ 2 ^25/2 ^~ 3 3 ^~ 

= — —x — c u x — —c u x + — x + —c u x + — c u x + — c u x + —x 
+ e + log (1 - e) + plog(l + u - e) - 2Jp 2 , (124) 

= Q - a; 3/2 +c u (l- ^c u ^j x 2 + '^c 2 u x b ' 2 + ^c 3 u x 3 + plog(l + u - e) 

+ e + log(l-e)-2 ( 5p 2 . (125) 

From (123) to (124) we expanded the first two brackets and from (124) 
to (125) we simplified. The dominant terms that does not go to zero as 
S — > are the terms with x and their sum is positive for c u < 1/5. Hence 
for fixed 7 there does not exist a <5 such that 2<f" max ^A" lax — e, 5, p*j < 0. 
Thus 

Prob (U(k, n, N; A) > W{5, p) - e) 

< poly (n, A™ ax - e) • exp [2n* max (A™ ax - e, 5, p)] , 

and as n — > 00 the right hand side of this does not go to zero. 

Now Part 1 and Part 2 put together shows that U 7 (S, p) is also a tight 
upper bound of U(k,n, N; A) with overwhelming probability as the problem 
size grows in the regime prescribed for U 7 (<5, p) in Theorem 4. 

□ 

3.3.2. The lower bound, Z~<(8 7 p 7 (5)) 
Proof. Lets also define 



31 



This implies that £ 7 (<5, p) = 1 — A™ m (<5, p) following from (8). Bounding £ 7 (S, p) 
above by -C 7 ((5, p) + e is equivalent to bounding A™ m below by A™ m — e. We 
first establish that for a slightly looser bound, with c; > 1/3, the exponent 

*min ^A™ m — e, S, pj is negative and then verify that when multiplied by n it 
diverges to — oo as n increases. We also show that for a slightly tighter bound, 
with q < 1/3, the exponent ^ m i n (A™ n + e, S, p^j is bounded from below by 

zero, and hence the bound £ 7 (<5, p) cannot be improved using the inequality 
(18) from [26]. We show, in two parts that for 7 > 70 fixed: 



1. 3 5 , e > and ci < 1/3 such that for S < S , * m i„ - e, 6, pj < 0; 

2. $ S , e > and ci > 1/2 such that for S < S , *,„in (A™ in + e, S, pj < 0, 
which are proven separately in the two parts as follows. 

Part 1: 



\ mm 



2* min (A™ - e, S, p) = 2H(p) + (1 - p) log (a^ 

+ plog(p) -p + 1 - (A- in -e) + - s U(Sp), (126) 



by substituting A™ in - e for A in (12). Let I := 1 - A™ in and bound the 
Shannon entropy functions from above using the first bound in (19) which 
gives 

2* mi „ (\™ n -e,S,p) 

< -2plog (p) +2p+{\-p) log [(1 - I) - e] + plogp - p + 1 - (1 - I) 
+ e-2plog(Sp) + ^(Sp), (127) 

= (l-p)log(l-Z-e)+plog(^y +Sp + l + e, (128) 
= I + log(l - I) + e + log (l - - plog (1 - I - e) 

+ / ?1 °g(^)+ 3 /'' ( 129 ) 
</ + -/- ^ 2 - ^ 3 + plog f -J-^] + 3p - p\og (1 - I - e) 



2 3 r a V<5V, 
+ log(l-e) + e, (130) 

< ~\l 2 - \l* + Plog (^3) + 3p - p\og (1 - I - e) - e + e. (131) 

We simplified from (127) to (128) and from (128) to (129) we expanded 
the first logarithmic term. From (129) to (130) we bounded 1/(1 — 1) below 
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and log(l — I) above using (52) and the third bound of (51) respectively. 
From (130) to (131) we simplified and bounded above log(l — e) using the 
first bound of (51). 

Let x — 2plog (^jr^sj + 6p which means I = ^fx — cix. We simplify (131) 

and replace the second two terms by \x and I in the first two terms by 
^fx — cix to get 



2* mi „ (\f*-e,5,p) 



1 2 1 3 1 

<-^(Vx-cix) --(Vx-Qx) + -x- plog(l-l-e) (132) 

= ~x + c lX 3 ' 2 - ^cfx 2 - ^x 3 ' 2 + Cl x 2 - c 2 x b ' 2 + ^cfx 3 + l -x 
-plog(l-Z-e), (133) 

= - Q - q) x 3 ' 2 + c t x 2 \c 2 x 2 c 2 x^ 2 + \c 3 x 3 

-plog(l-Z-e). (134) 

From (132) to (133) we expanded the first two brackets and from (133) 
to (134) we simplified. Substituting 1/ [7 log (1/5)} for p in the expression 
for x we have x = 4/7 + g(p) where g(p) — 6plog (1/p) + 6p and goes to 
zero with 5. We make the same argument as in Part 1 of the proof for 
IP (5, p-f(5)) in Section 3.3.2, that is for 7 > 4 we can define Si such that 
for 5 < Si, x < 1 and we can upper bound x 3 by x 2 since x 2 > x 2+: > for 
j > when x < 1. The last term in (134) goes to zero with 5, so we can 
define 62 such that for 5 < 62 we can bound this term above by x 2 which is 
a constant. We split the first term of (134) into half and drop the two cf 
terms because they are negative. Let fi(ci) = q + cf/3, which is positive 
for all Q > 0, using the above we upper bound (134) as follows. 



2vf min (A™ in -e,<5,p) 



2 V3 J 2 \ 3 



-\(\- <*) * 3/2 -Ul-<<) * m + flMx 2 + x 2 , ( L35) 
. -^=[--r l )--(--n)x 3 / 2 + [l + f l (c l )}x 2 . (136) 





\ — 1 




(H 


-2 1 





From (135) to (136) we use the fact that —x 3 l 2 < S/^/j 3 as shown in 
Section 3.3.2. For (136) to be negative all we need is for cj < 1/3 and the 
sum of the last two terms to be non positive, that is: 



4(H r,/2+|1+/ ' fa)1 ^° - ^Wrlk}- 

(137) 

Let's define £3 such that for 5 < 5 3 (137) holds; since £ is a decreasing func- 
tion of S^ 1 for fixed 7 there exist a unique £3. We set So = min (Si, 62, S3) 
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and conclude that if c/ < 1/3, for fixed 7 > 70 = 4 and e > when S < So 
as S — > (136) will remain negative and 2* m i n ^A™ ln — e, <5, p^ < 0. 

Having established a negative bound from above and the <5 for which 
it is valid, it remains to show that n ■ 2vE f min ^A™ m — e, 5, p^j — > —00 as 

(k,n,N) — > 00, which verifies an exponential decay to zero of the bound 
(18) with n. This follows from the first term of the right hand side of 
(136) giving a concluding bound — n ■ 4 (1/3 — c/) / ^/7^- For 7 > 70 and 
8 < So therefore 

Prob (l,(fc, n, N; A) > I' 1 (S, p) + e) 

<poly (n,A™ in + e) • exp 
The right hand side of which goes to zero as n — > 00. 



An fl \ 



Part 2: 



2* mi „ (a™ + e, <5, p) = 2H(p) + (1 - p) log (a™ + e) 

+ plog(p) - p + 1 - (Af n + e) + |h(J P ), (138) 



by substituting A™ m + e for A in (12). Let / := 1 - A™ m and bound the 
Shannon entropy function from below using the second bound in (19) to 
give 

2* mi „ (\f n + e,S,p) 

> 2 [-plogp + p - p 2 ] + (1 - p) log [(1 - I) + e] + plogp - p 
+ 1 - (1 - - e + - & [-p log (Sp) + Sp- S 2 p 2 ] , (139) 

= -2plogp + 2p- 2p 2 + log (1 -l + e) -plog(l - I + e) + plogp - p 
+ 1 - 1 + I - e - 2plog {Sp) + 2p- 25p 2 , (140) 

1-1J '""^ ' 1 w 1 ''^"(^v 

+ 3p-2(l + <5)p 2 , (141) 

>-i- h 2 + i + piog (j^) +' 3 p + lo § (! + £ ) - 6 

- plog(l -l + e)- 2(1 -5)p 2 , (142) 



= log (1 - + log 1 + - + Z-e-plog(l-Z + e)+plog 



>_ 2 2 +pl0g ( v 5V J + 3p + £ ~ 2 e - £ -P 1 °g( 1 - / + £ ) 
- 2(1 - 5)p 2 . (143) 
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From (139) to (140) we expanded brackets and simplified. From (140) to 
(141) we expanded log(l — I + e) and simplified. From (141) to (142) we 
bounded from below 1/(1 — I) using (52) and using the bound of (61) we 
also bounded from below log(l — I). Then from (142) to (143) we simplified 
and bounded from below log(l + e) using (42). 

Let x = 2p log (^gr^j + 6p which means I = sfx — c\x. We simplify (143) 

and replace the second two terms by x/2 and I in the first two terms by 
y/x — qx to get 



+ e,5, p) 



= ci 



2* min (A™ n 

1 2 1 3 1 

> (Vx-cix) - - (y/x-cix) + -x - plog(l -l + e) 

-2(l-5)p>- l -e\ (144) 
= ~x + Qx 3 / 2 - X -c 2 x 2 - l -x z l 2 + ^qx 2 - ^c 2 x 5 / 2 + icfx 3 + X -x 
-plog(l-Z + e)-2(l-% 2 -ie 2 , (145) 
I) ^ 2 + \ Cl (3 - q) x 2 \c^ 2 + icfx 3 - plog (1 - / + e) 

-2(l-8)p 2 - 1 -e 2 . (146) 

From (144) to (145) we expanded the first two brackets and simplified 
from (145) to (146). The dominant terms that does not go to zero as 
5^0 are the terms with x and their sum is positive if q > 1/2 and 
x < 1. We established in the earlier parts of this proof of Theorem 4 that 
if 7 > 4 we will have x < 1 as <5 — > 0. Hence we conclude that for fixed 
7 > 7q = 4 and e > there does not exist a <5 such that (146) is negative 

and 2* min (A™ in + e, 5, pj < as 5 0. Thus 

Pro6 (L(fc, n, iV ; A) > {5, p) - 

< poly (n, A^ in + e) • cxp [2n* min (A^ in + e,6,p)], 
and as n — > oo the right hand side of this does not go to zero. 

Now Part 1 and Part 2 put together shows that £ 7 (<5, p) is also a tight bound 
of L(k, n, N; A) with overwhelming probability as the sample size grows in the 
regime prescribed for £ 7 ((5, p) in Theorem 4. 

□ 

3.4- Corollary 5 

Proof. We prove Corollary 5 in two parts, first proving the case for IX 7 (S, p 7 (<5)) 
and then that of £ 7 (<5, Pj(S)). 



35 



Part 1: From (7), for p = p y {8) = (7 log (})) , we have 

2plog 



W(S,p^5)) = W2plog 



J 2 ^ 



6p + c u 



1 



= \/2plog 



8 2 p' ; 



Qp + 2c„plog 



(5 2 p 3 

1 



+ 6/9 



. (147) 



S 2 p 3 



+ 6c u p (148) 



= W4plog 



6p log 



6p + 4c„plog 



+ 6c„plog ( - ] +6c u p, 



V 7 
+ 6c M p. 



4c u 



+ 6p log - + 6p H h 6c„p lo, 



7 



(149) 



(150) 



From (147) to (148) we expanded the square brackets while from (148) to 
(149) we separated the terms explicitly involving S from the rest. From 
(149) to (150) we substituted 1/ [7 log (1/5)] for p in the terms explicitly 
involving S and simplified. 

Now using the fact that lim^o pl°g (Vp) = and lim^o p = we have 
lim^(5,p 7W) = A + ^, 

hence concluding the proof for IP (5, p 7 (<$)). 

Part 2: From (8), for p = p 7 (5) = (7 log (|)) 1 , we have 

/ 1 



^(5,p 7 (5)) = 4 2/9 log 



5V 



+ 6p - C; 



2plog 



/ 2p log 



5 2 p 3 



+ 6p - 2c;/9log 



1 



+ 6/9 



(151) 



6c ; p, (152) 



4plog - + 6plog - + 6p - 4c/plog - 



6c; p, 



- 6c//9log 



+ 6/9 log 



7 
6c;p. 



6p- — - 6c/plog 

7 



(153) 



(154) 
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From (151) to (152) we expanded the square brackets while from (152) to 
(153) we separated the terms explicitly involving 6 from the rest. Then 
from (153) to (154) we substituted 1/ [7 log (1/5)] for p in the terms ex- 
plicitly involving 5 and simplified. 

Now using the fact that lim^o pl°g (1/p) =0 and lim^o P = we have 
lim^p 7 (5)) = A_^, 

hence concluding the proof for IP (5, p 7 ((*>))■ 
Part 1 and Part 2 combined concludes the proof for Corollary 5. □ 
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